17.4 Gibbs SamplingΒΆ

Draw sample from distribution q(x): updating \(x \leftarrow x' \sim T(x'|x)\)

  • Problem: how to ensure that q(x) is a useful distribution

  • Solution:

    • Derive T from a goven learned model \(p_{model}\)
    • Directly parameterize T and learn it, so that its stationary distribution implicitely define the \(p_{model}\) of interest.

In DL, we commonly use Markov chain to draw samples from an energy-based model defining a distribution \(p_{model}(\vec{x})\). In this case, we want the \(q(\vec{x})\) for the Markov chain to be \(p_{model}(\vec{x})\). To obtain the desired q(x), we must choose an appropriate \(T(x'|x)\)

Gibbs sampling: sampling from \(T(x'|x)\) is accomplished by selecting one variable \(x_i\) and sample it from \(p_{model}\) conditioned on it neighbor in the undirected graph G defining the structure of the energy based model. We can also sample several variables at the same time as long as they are conditionally independent given all their neighbors.

e.g Restricted Boltzman Machine

../../_images/Figure16.141.PNG

all the hidden units of an RBM maybe sampled simultaneously because they are conditionally independent from each other given all the hidden units. Gibbs sampling approaches that update many variables simultaneously in this way are called blocled Gibbs sampling.